CRESST REPORT 813 



IMPACT OF INCENTIVES ON THE USE OF 
FEEDBACK IN EDUCATIONAL VIDEOGAMES 

MARCH, 2012 



Girlie C. Delacruz 



National Center for Researc 

on Evaluation, Standa|:cls, & Student Testin 



Impact of Incentives on the Use of Feedback in Educational Videogames 



CRESST Report 813 



Girlie C. Delacruz 

CRESST/University of California, Eos Angeles 



Mareh, 2012 



National Center for Researeh on Evaluation, 
Standards, and Student Testing (CRESST) 
Center for the Study of Evaluation (CSE) 
Graduate School of Education & Information Studies 
University of California, Eos Angeles 
300 Charles E. Young Drive North 
GSE&IS Bldg., Box 951522 
Eos Angeles, CA 90095-1522 
(310) 206-1532 




Copyright © 2012 The Regents of the University of California. 

The work reported herein was supported by Center for Advance Technology in Schools (CATS), PR/Award 
Number R305C080015, as administered by the Office of Educational Research and Improvement, U.S. 
Department of Education. 

The findings and opinions expressed in this report are those of the author(s) and do not necessarily reflect the 
positions or policies for the Center for Advance Technology in Schools (CATS), the Institute of Education 
Sciences (lES), or the U.S. Department of Education. 

To cite from this report, please use the following as your APA reference: DelaCruz, G. C. (2012). Impact of 
incentives on the use of feedback in educational videogames. (CRESST Report 813). Los Angeles, CA: 
University of California, National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST). 




TABLE OF CONTENTS 



Abstract 1 

Introduction 1 

The Importance of Feedbaek for Learning 2 

Inereasing the Use of Feedbaek in Teehnology-Based Learning Environments 

via Incentives 2 

Rubries of Seoring Rules to Promote Clarity of Expeetations 3 

Summary 4 

Methodology 4 

Research Questions 4 

Researeh Design 4 

Save Patch Instruetional Eeatures 7 

Proeedures 8 

Analyses 9 

Pooling the Points-Only and No Seoring Rules Information Data 9 

Results for Math Aehievement 9 

Conelusion 11 

Referenees 13 



iii 




IMPACT OF INCENTIVES ON THE USE OF FEEDBACK 



IN EDUCATIONAL VIDEOGAMES 

Girlie C. Delacruz 

CRESST/University of California, Los Angeles 



Abstract 

Educational videogames can be designed to provide instructional feedback that is 
responsive to specific actions. However, existing research indicates that students tend 
to ignore videogame feedback and subsequently use less effective help-seeking 
strategies. Research on help-seeking in learning environments has primarily focused 
on the role of cognitive factors, the nature of the help, or issues of timing and 
frequency. There is a noticeable gap in understanding regarding how to motivate and 
increase the use of feedback for improved learning. Using a pre-algebra videogame, 
this study examined the relationship between an incentive to use feedback and math 
achievement. A randomized-control design was employed, which compared learning 
outcomes of students who received the incentive to those who did not. Results 
indicated that students given the incentive to use feedback had significantly higher 
normalized change scores on math items {d = .53), with stronger effects for students 
with low academic intrinsic motivation (J = .88 - 1.17). 

Introduction 

A key benefit of games for learning is that they can be designed to provide instructional 
feedback responsive to specific student actions. Tailored hints and feedback, or supporting 
information accessible via general help menus, is the type of help that is often used in help- 
seeking studies. However, the research on help-seeking indicates that when students reach an 
impasse, they either use ineffective help-seeking strategies, or avoid seeking help altogether, as 
reviewed by Aleven et al. (2003). 

To understand the relationship between help-seeking and learning, researchers have 
primarily studied the role of cognitive factors (e.g., self-efficacy), the nature of the provided help 
(e.g., context-specific versus generalized principles), timing and frequency (e.g., on demand or 
system-initiated), and how the process of help seeking can be taught explicitly. There is a 
noticeable gap in understanding regarding how to motivate students to effectively seek help and 
to use feedback. 

Consequently, this study focused on learning games where game play was directly 
integrated and linked with its corresponding academic content. If game play is to require the use. 
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demonstration, and evaluation of the application of knowledge or skills in a particular domain, 
students may need to know the criteria underlying their progress or lack thereof. Communication 
of a game’s scoring rules can be leveraged to make assessment criteria more explicit and 
transparent. 

This paper describes a study that examined the impact of incentives to access feedback, 
combined with different degrees of game explanation rules, to math achievement. The learning 
platform was an educational videogame designed to teach students about fractions. 

The Importance of Feedback for Learning 

Feedback related to progress in learning tasks can generally have a positive effect on 
learning (Nyquist, 2003). There are certain conditions that make it effective — specifically, 
feedback that supports clear understanding of the learning goals or objectives (and the criteria 
that define good quality), methods for students’ to relate his or her performance to the goals, and 
a clear path or paths to achieve the goals. (Bangert-Drowns et ah, 1991; Hattie & Temperley, 
2007; Kluger & DeNisi, 1996). 

However, the mere presence of feedback, hints, or available help is insufficient for 
learning. Review of the research on the use of feedback in technology-based environments 
indicates that students rarely access available feedback voluntarily. This is problematic as the 
lack of compliance with the treatment greatly reduces any potential effect on the outcomes. For 
example. Nelson (2007) compared different levels of feedback on student achievement in an 
immersive learning environment. Results from the study indicate that most students did not 
access the feedback. Also, between students who were provided extensive feedback and students 
who were given moderate feedback, there were no statistical differences on the frequency of 
accessing the hints. In a study that examined the effect of providing user-initiated feedback via a 
pedagogical agent. Van Eck and Dempsey (2002) also reported low levels of student access to 
the feedback. The authors concluded that further research is essential to figure out how to 
promote its use. 

Increasing the Use of Feedback in Technology-Based Learning Environments via 
Incentives 

Given the historic barriers, it is not surprising that students avoid seeking help. Research 
conducted in classroom settings, for example, suggests that some students perceive help-seeking 
as a public sign of failure and therefore a social stigma (Karabenick & Newman, 2006; Puustinen 
et al, 2009; Ryan & Pintrich, 1997). Further, in game-based settings, accessing feedback slows 
the game down, and some games have speed of play as a basis of advancement. In some 
technology-based environments, accessing feedback or available hints is associated with low 
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proficiency and subsequently discouraged. For example, when a hint is given to a student in the 
PACT Geometry tutor, the student’s visible -skill bar” decreases (Aleven & Koedinger, 2000). 
In some games, students are penalized when accessing help, (i.e., students lose points when a 
hint is accessed). 

The design approach used in this study adopted the perspective that incentives can 
communicate what is valued within a given context and promote desired behavior. Rewards and 
incentives are, after all, a method for signaling those actions or behaviors that are encouraged or 
discouraged by a particular community. Conveying these expectations is a key process through 
which individuals learn how to participate in learning and gaming communities (Rogoff, 1900; 
2003). The use of an incentive to access feedback may be one potential approach, especially as 
findings from studies using tasks with initial low interest suggest that incentives may be 
beneficial to learning. (Cameron, 2001; Cameron & Pierce, 1994). Moreover, in video games, 
incentives are intrinsically tied to performance and are often a permanent aspect of the activity. 
In fact, game designers argue that incentives (either in the form of rewards or punishment) are 
essential to the fun or sometimes the compulsion of the experience (Fullerton, 2008; Koster, 
2005; Schell, 2005). At best, an incentive can reverse the association between help-seeking and a 
perception of failure, signaling to the student that seeking feedback is a valuable act leading to 
proficiency. 

Rubrics of Scoring Rules to Promote Clarity of Expectations 

In formal learning settings, scoring rubrics (or codified scoring rules and corresponding 
score values) have been used to improve assessment clarity by making explicit what constitutes 
good performance. There have been numerous studies on teachers’ use of rubrics, typically for 
student-constructed responses. Most of the research has examined how best to train teachers to 
effectively use rubrics to increase the reliability and validity of scoring performance assessments 
(Baker et ah, 1995). Opportunities for student use of rubrics to improve learning appears logical, 
although only a few studies have examined this idea directly (Brown et ah, 2004; Jonsson & 
Svingby, 2007; Sadler & Good, 2006). 

When the scoring rules are appropriately tied to academic progress, providing an 
explanation of the game’s scoring rules functions as a rubric in a game for learning. The scoring 
rules make explicit the stated learning objectives of the game as well as the criteria used to 
evaluate performance. The scoring rules can direct attention to what and how responses are being 
scored which may (a) make more explicit the learning objectives or goals of the game, (b) clarify 
the criteria of performance (i.e., what -counts”), and (c) support the development of self- 
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assessment of performanee to determine when additional help is neeessary. This information ean 
be provided prior to game play to guide performanee, or as a eontext for elaborated feedbaek. 

Summary 

The literature review indieates that for learning environments to be effective, students need 
to use feedback in order to circumvent the ineffectiveness of trial-and-error learning. In this 
study, two aspects of design features were examined in game-based environments on math 
achievement, (a) incentivizing the use of feedback, and (b) different levels of making 
performance criteria explicit through the explanation of scoring rules. 

Methodology 

The following section will describe the research question addressed by the study, the 
research design, descriptions of the treatment conditions and dependent measures used in the 
study, and data analyses. 

Research Questions 

In a game designed to teach the addition of rational numbers: 

1. What is the effect of providing an explanation about the scoring rules on (a) math 
achievement scores and (b) game performance? 

2. Do incentives to seek additional feedback affect (a) math achievement scores and (b) 
game performance? 

3. Does (a) lack of information or (b) providing an incentive affect the frequency with 
which students voluntarily access feedback? 

Research Design 

Data was collected from 112 students in fourth to sixth grades in after-school contexts. A 
randomized-control, 1x4 design was used in this study. There were four treatment conditions, 
which are described in the following section. Within participating after-school programs, each 
participant was randomly assigned to one of the four conditions. 

Treatment Conditions. Table 1 contains descriptions of the four treatment conditions used 
in the study with regard to the amount of rules explanation and the incentive to use feedback. 
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Table 1 



Description of Treatment Conditions Used in the Study 





Amount of information player was given regarding game’s scoring 

rules 


After a mistake 
was made, player 
was given an 
incentive to 
access additional 
feedback help 


Treatment condition 


Explanation of 
scoring rules 
given prior to 
game play 


Scoring 
explanations 
available in 
general help 
menu 


Explicit feedback 
about when and 
how many points 
are earned and 
lost 


Math 
elaborated 
feedback 
of scored 
event 


No scoring 
information (n = 28) 


0 


0 


0 


0 


0 


Points-only feedback 
(n = 20) 


0 


0 


X 


0 


0 


Explanation of 
scoring rules (n = 22) 


X 


X 


X 


X 


0 


Rewarding help 
seeking (n = 28) 


X 


X 


X 


X 


X 



Experimental Materials: Save Patch. The game used in the study targeted two key 
mathematies ideas: (a) only identical units can be added to create a single numerical sum, and (b) 
the size of a rational number is relative to how one whole unit is defined. 

The objective of the game was to help the game character (Patch) jump over obstacles (e.g., 
spikes, lava, quicksand) and move from block to block to reach the last block (the final 
goal). In order to fulfdl these objectives, students needed to compute the distance of the jump, 
place trampolines on the blocks, and add enough coils to the trampolines to make Patch bounce. 
The size of the coil determined how far Patch would bounce. For example, a one -half unit coil 
would cause Patch to jump over a one-half unit interval. 

The first part of the game required the student to determine the size of the intervals of the 
grid. The intersection of vertical red bars indicated the boundaries of the whole unit. The green 
dots broke up the whole unit into intervals (see Figure 1). 
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Figure 1. The intersection of the vertical bars depicted the boundaries of the 
whole unit. The green dots broke up the whole unit into intervals. 

Once the students figured out the unit size of the spaces on the grid, they had to add 
together the correct number of coils that will span the distance to jump over. For example, in 
Figure 2, to get safely from one block to the next. Patch needs to jump over three one-third-unit 
intervals. This means that to successfully make it to the next block, the trampoline must contain 
three one-third-unit-sized coils. If the trampolines did not contain the correct number or size of 
coils necessary to get Patch to the next block. Patch exploded into feathers and the students were 
allowed to replay the level. To reinforce the idea that only fractions with like denominators can 
be added together, the student could not combine coils that have different unit sizes. If a student 
had a trampoline with a one -third coil on it and tried to add a one-sixth coil, the one-sixth coil 
would not go onto the trampoline. 
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Figure 2. Jump distance equals total number of spaces between blocks. The 
number of coils to add to a trampoline equals the jump distance. 

Save Patch Instructional Features 

Tutorials. The tutorial design was meant to first contextualize the math concept within the 
game, and how the math concept relates to math in general. The tutorial information was 
presented in both written text and as guided interaction. 

Scored events during game play. Across all of the conditions, three events were chosen as 
key points where performance would be evaluated because they mapped onto the learning 
objectives of the game. Points were earned any time the following event occurred: (a) student 
used coils that were the correct unit size for the grid, (b) student added together coils with like 
denominators, and (c) student successfully completed the level. Points were lost when any of the 
following events occurred: (a) student used coils that were not the correct unit size for the grid, 
(b) student attempted to add coils with unlike denominators together, and (c) student failed to get 
Patch to an intermediate goal. 

Feedback. All of the players in each condition received feedback after three events. This 
feedback was in the form of the knowledge of results without elaboration or explanation. When a 
player tried to add a coil to a trampoline that had a coil with unlike denominators, the coils 
would not combine on the trampoline. Also, Patch exploded into feather when the trampoline 
had the wrong: (a) coil size and (b) amount of coils needed to cover the jump distance. When 
Patch jumped from block to block without exploding, this indicated that the player placed the 
correct quantity of coils needed to make the jump. 
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General help menu. All of the players in each condition had access to a general help menu 
at any time throughout the game levels. The topics that were included in the general help menu 
provided information on game mechanics (e.g., how to make Patch move in different directions) 
as well as instructional information such as how to choose the right-sized coil and how to add 
coils of different sizes. 

Feedback help. After the second consecutive mistake, players in each condition were 
given an opportunity to access additional feedback by clicking on a button that read, -€lick here 
for help.” The additional feedback provided elaborated explanations and hints that were designed 
to assist the player with repairing the mistake. 

Procedures 

An average of about 90 minutes was spent on the study at each site. The basic timing and 
order of the task were the same at each site. It took students about 10 to 15 minutes to complete 
the test, although the time necessary to complete the test never exceeded 15 minutes. 

Every effort was made to ensure that each student played the game for at least 30 minutes. 
However, the amount of time for game play varied because many students departed for home 
early. Students who did not play the game for at least 20 minutes were dropped from the 
analyses. 

After playing the game, the students were given the posttest and the 
demographic/motivation survey. Students were told that some of the questions that appeared on 
the pretest would be on the posttest, but that the information they learned from playing the game 
might be useful to answer the posttest questions. 

Table 2 



Description of Dependent Measures Used in the Study 



Math achievement 


Game performance 


Total proportion of times additional feedback accessed 


Pretest: 31 items 


Number of: 


Overall in the game 


Posttest: 44 items 


coils added together (both 


After coils with different denominators were added 


(includes items in 


common and unlike 


together 


game context) 


denominators) 
wrong-sized unit coils used 
resets 

failed attempts 


When a wrong-sized unit coil was used 
Failed attempt 




Analyses 



Pooling the Points-Only and No Scoring Rules Information Data 

The purpose of pooling the data from the points-only and no seoring rules eonditions was 
to ereate a eondition that represented a minimal seoring information group. Two independent 
samples t-tests were eomputed to eompare the groups on the math aehievement measures to 
ensure they did not differ signilieantly. There were no signifioant differences between the two 
groups on the posttest, t(48) = .58, p = .57, transfer items, t(48) = 1.19, p = .24, or the game 
context item scores, t(48) = 1.03, p = .31. Therefore, for the purposes of testing the three 
hypotheses of the study, the data from students in the points-only and no scoring rules groups 
were pooled together and henceforth referred to as the minimal scoring information group. Three 
sets of orthogonal planned comparisons were conducted to examine if the data supported the 
hypotheses of the study. 

Results* for Math Achievement 

Table 3 contains a summary of results for the math achievement scores. For the overall 
sample, students who were given the explanation of the scoring rules plus the incentive had 
significantly higher normalized change scores, compared to students who were given minimal 
scoring information, better had the results indicated that compared to students who received 
minimal scoring information, students who were provided both explanation of the scoring rules 
and an incentive to seek additional feedback had significantly higher normalized gain scores on 
the math assessments (d = .53) and better game performance (d = .99 - 1.09), with stronger 
effects for students with low academic intrinsic motivation (d = .88 - 1.17). 



' For an extended discussion of the results of the study, please see Delacmz (2011). 
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Table 3 



Summary of Results for Math Achievement Scores 



Sample of students 


Posttest scores 


Gamelike items 


Normalized change scores 


Overall sample 


No group differences 


For students with low prior 
knowledge, scores were higher 
for explanation of scoring 
rules -1- incentive group 

For students with high pretest 
scores, scores were higher for 
explanation of scoring rules 
group. 


Minimal scoring < 
Explanation of scoring 
rules -1- incentive (d= .53) 


Low self-efficacy 


Minimal scoring 
information group < (a) 
Explanation of scoring 
rules and (b) Explanation 
of scoring rules -I- incentive 

(t/=.88) 


Highest for students in the 
explanation of scoring rules + 
incentive group {d = .65) 


No group differences 


Lower math self- 
concept 


No group differences 


Highest for students in the 
explanation of scoring rules -l- 
incentive group (c/ = .89) 


No group differences 


Lower preferences 
for cooperative 
learning 


No group differences 


Highest for students in the 
explanation of scoring rules -t- 
incentive group {d = .97) 


No group differences 


Males 


No group differences 


No group differences 


Highest for students in the 
explanation of scoring 
rules -1- incentive group (d 
= .93) 



Game performance. Students who were given the explanation of seoring rules plus the 
incentive added fewer coils with unlike denominators than the students in the other conditions {d 
= .61 - 88). Student who were given minimal scoring information had to reset the level more 
times and had more failed attempts than students in the other conditions {d = .99 - 1.09). 

Results for voluntarily accessing feedback. Students given the incentive accessed the 
feedback less frequently and spent the least amount of time on the feedback screens than the 
other students. However, when feedback was accessed, compared to the other students, they 
would solve the game level more quickly and with fewer mistakes. Furthermore, upon the 
occurrence of the same mistake, students given the incentive sought additional information in the 
help menu on the same topic more often than students not receiving the incentive. 
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Conclusion 



Overall, findings suggest that when designing games for learning, it is important both to 
make performance criteria (in this case, the scoring rules of a game) explicit to students and 
motivate students to use provided feedback through the use of incentives. There are two potential 
explanations for the positive effect of the combined incentive and explanation of rules. First, 
providing the rationale for the scoring rules may clarify the criteria of performance, (i.e., what 
-counts”). Second, the incentive signals that the use of feedback is valuable and may increase the 
likelihood that students will engage in self-directed help-seeking and deeper engagement. 
Findings from this study suggest that it is important both to make assessment criteria explicit to 
students and to find ways to motivate students to use provided feedback through the use of 
incentives. 
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